Skip to content

[2/2] feat: embed component WIT in the compiled Miden package - #1248

Draft
greenhat wants to merge 32 commits into
i1302-MPC-unque-by-inputsfrom
wit-in-package
Draft

[2/2] feat: embed component WIT in the compiled Miden package#1248
greenhat wants to merge 32 commits into
i1302-MPC-unque-by-inputsfrom
wit-in-package

Conversation

@greenhat

@greenhat greenhat commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Close #345
Close #1298

This PR is stacked on #1306 and should be merged after it.

Summary

The component WIT generated by #[component] is now embedded in a Wasm custom section (rodata,miden_wit), carried through the compiler pipeline, and stored as a wit section of the compiled .masp. The #[account(...)], sibling #[component(pkg::Iface)], #[note], #[tx_script], and generate! macros read dependency WIT from the dependency's compiled package instead of from target/generated-wit/, so the wit = "..." path metadata in miden-project.toml is gone and prebuilt .masp file dependencies are self-contained.

Previously every consuming project carried wit entries pointing into another crate's target directory, needed only because WIT selection and binding resolution read WIT from different places — the trap behind the sibling "package not found" failures (the augment_missing_sibling_wit workaround is deleted). Now both read the same single source.

Details

  • Pipeline: both out-of-band payloads (account component metadata, WIT) travel in one PackageSections carrier struct from the frontend to assembly, where they are attached as package sections — a future payload needs no per-stage threading.
  • Manual authoring: a bare miden::generate!() over a local wit/ directory embeds the WIT too, provided it is a single self-contained file; anything else is skipped, and consumers get the accurate "does not embed component WIT" error.

@greenhat
greenhat marked this pull request as ready for review July 6, 2026 06:43
@greenhat
greenhat requested a review from bitwalker July 6, 2026 06:43
@greenhat

greenhat commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased and ready.

@bitwalker bitwalker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to retain the wit field in miden-project.toml - otherwise there will be no way to manually provide WIT for a package that does not contain it. It is an edge case of course, but unfortunately a necessary one to cover at this stage I think.

This only caught my eye because I just implemented WIT embedding in the branch containing the language I'm building, and there I have it so that the embedded WIT is the default, but the wit key in miden-project.toml is checked as a fallback/escape hatch.

I'm hoping to open the PR that introduces the language this weekend/early next week - just depends on how much time I have, but since linking between Rust-compiled packages and those compiled from this new language is an explicit goal of mine, I'm keeping a close eye on anything that affects how we link/bind packages.

@greenhat

Copy link
Copy Markdown
Contributor Author

I think we need to retain the wit field in miden-project.toml - otherwise there will be no way to manually provide WIT for a package that does not contain it. It is an edge case of course, but unfortunately a necessary one to cover at this stage I think.

Done in 2d7507a. In the case where the WIT is both present in the Miden package and in the toml file, I raised an error.

@greenhat
greenhat requested a review from bitwalker July 10, 2026 11:14
@greenhat

Copy link
Copy Markdown
Contributor Author

Rebased and ready.

@greenhat

Copy link
Copy Markdown
Contributor Author

Rebased (ported) and ready.

@bitwalker

Copy link
Copy Markdown
Collaborator

@greenhat I've followed up on #1300 with how I think we have to proceed with regards to dependency management, perhaps we should make those changes as part of this PR, or make them in a separate PR and rebase this PR on that one? Dependency management is so core to the way the SDK macros work, that I'd rather solve it first, than to merge more hacks that sidestep the fundamental problem.

@greenhat

greenhat commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

@greenhat I've followed up on #1300 with how I think we have to proceed with regards to dependency management, perhaps we should make those changes as part of this PR, or make them in a separate PR and rebase this PR on that one? Dependency management is so core to the way the SDK macros work, that I'd rather solve it first, than to merge more hacks that sidestep the fundamental problem.

I agree.
This PR is already quite big. I'll do it in separate PR(s) and rebase this one afterwards.

@greenhat
greenhat marked this pull request as draft August 3, 2026 10:06
greenhat added 13 commits August 6, 2026 15:53
The filesystem package cache at target/miden/packages persisted across builds and keyed files by package name alone. When build inputs changed (dependency set, versions, pins, or the compiler itself), leftover .masp files from older builds could satisfy the SDK proc-macro lookups through MIDENC_PACKAGE_CACHE and bake stale FPI procedure roots into generated code, failing only at transaction execution (#1302, surfaced by #1300).

Derive the cache path as target/miden/packages/<fingerprint>, where the fingerprint hashes the compiler version and revision, the build-relevant options, and the project's recursive manifest closure (including resolved dependency schemes and preassembled package contents). Session memoizes the fingerprint, so the package registry and the MIDENC_PACKAGE_CACHE variable handed to nested cargo builds keep agreeing on one derivation.

Registry construction now also prunes stale midenc-owned cache entries (fingerprint directories and legacy flat .masp files) next to the current directory. The pruning is load-bearing: the macros track their package reads with include_bytes! dummies, so a surviving stale directory would keep cargo reusing a stale macro expansion. Deleting it forces re-expansion against the fresh cache.

The fingerprint intentionally excludes Rust sources and lockfiles: every run rewrites each resolved package into the cache before its consumers expand, and content changes at a stable path already invalidate consumers through the include_bytes! tracking.

The cargo-miden integration tests that asserted the flat packages/<name>.masp layout now locate the build's single fingerprint directory instead.
…erprint gaps

A pre-submit review of the fingerprinted package cache found one race and several hardening gaps. The prune deleted every sibling fingerprint directory unconditionally, so two concurrent builds of one project with different inputs (debug and release builds of the same checked-in example, as the test suite itself arranges) could delete each other's live cache between a package write and the consuming macro's read.

Each build now holds an exclusive advisory lock on a .build-lock file inside its fingerprint directory for the registry's lifetime, and the prune deletes a sibling only when its lock is free or absent. A live directory is skipped; a lock that cannot be verified is left in place with a warning, since deleting an unverifiable cache risks reviving the stale-expansion bug the prune exists to prevent. Pruning is also refused entirely when the target path is not fingerprint-named, so external callers of the public constructor cannot sweep an arbitrary parent directory, and failed removals of owned entries are logged at warn with their consequence.

The fingerprint gains two inputs that escaped it: the inherited RUSTFLAGS environment (composed into every nested cargo build) and the containing workspace's root manifests (member manifests do not change when workspace-level fields do). Moved git branches remain outside the fingerprint by design, now documented. The fingerprint format is defined once and shared by the producer, the prune recognizer, and their tests; record_options destructures Options exhaustively so a future field must be explicitly classified as fingerprinted or ignored; link libraries contribute their declared identity instead of a redundant package load; and the design rationale that previously lived outside the tree is captured in module and function docs.

The FPI macro diagnostic for the cache-lookup branch now names the searched MIDENC_PACKAGE_CACHE directory and the expected package file names instead of an empty candidate list and a profile-directory hint that branch never consults.

New tests pin the liveness behavior (a locked sibling survives, an unlocked one is pruned), the misuse guard, the fingerprint walk's cycle guard and degradation markers, and — end to end — the invalidation contract itself: rebuilding after a dependency source change keeps the same cache path but replaces the FPI procedure root baked into the consumer's assembly.
The stale-expansion story rested entirely on pruning: a consumer's cached macro expansion was re-expanded only because its include_bytes! target vanished. Pruning is best-effort, so every failure path — a live locked directory that deliberately survives, a failed removal on a restrictive filesystem, a directory recreated by a still-running old-input build — left the original stale-roots bug reachable.

Emit const _: Option<&str> = option_env!("MIDENC_PACKAGE_CACHE") into every FPI expansion, next to the existing include_bytes! constants. The variable's value carries the fingerprinted cache path, so rustc records it in the consumer's dep-info and Cargo re-expands the macro whenever the fingerprint rotates, even when a stale directory survives on disk. The same mechanism already invalidates cached expansions for MIDENC_EMIT_WIT. include_bytes! keeps covering content changes at an unchanged path.

Pruning and locking thereby demote from correctness-critical to defense in depth: they remove legacy flat files whose expansions predate this tracking, keep the cache parent bounded, and prevent transient mid-build file loss. The prepare_filesystem_cache doc is updated to say so.
The pre-submit review found four defects around cache preparation. A build that failed to create its own cache directory still swept every sibling; an identical-fingerprint contender ran unprotected after observing WouldBlock, so its directory could be deleted mid-build once the first holder exited; the lock file lived inside the directory it protected, leaving create-before-lock and unlock-before-delete windows; and the public constructor pruned the parent of any 16-hex-named path, so an arbitrary caller-supplied location could have unrelated siblings deleted.

Locks now live outside the deletable directory as packages/<fingerprint>.lock and are acquired before the directory is created. Builders hold the lock shared, so any number of identical-input builds stay protected at once; pruning demands the exclusive lock and holds it while remove_dir_all runs, then removes the orphaned lock file after verifying no builder acquired it. The residual close-to-unlink race is documented as accepted: option_env!(MIDENC_PACKAGE_CACHE) in FPI expansions is the correctness boundary, and pruning is defense in depth. Preparation now stops before any deletion when the cache or its parent cannot be created, and the destructive sweep runs only for paths in the owned miden/packages/<fingerprint> layout.

Registry insertion also detects a same-name, same-version, different-digest conflict before touching disk, so a rejected package no longer overwrites the cache file that the accepted in-memory package no longer matches. Accepting paths still rewrite their file on every run, which the content self-heal relies on. Legacy flat cache entries are matched case-insensitively via Package::EXTENSION, so a Legacy.MASP leftover no longer survives the sweep on case-insensitive filesystems.

Prune failures keep their consequence in the message and now name the actual parent directory; the log-only nature of cleanup reporting is documented on the constructor.
Three inputs escaped the fingerprint or degraded it silently. A bare relative --manifest-path (e.g. plain Cargo.toml) produced an empty project directory, so the manifest walk recorded a load failure and never visited path dependencies; the derivation now absolutizes the locator against the session's configured working directory. A workspace member dependency was classified by file extension instead of being resolved through the loaded workspace; it now resolves via get_member_by_relative_path like the canonical resolver. RUSTUP_TOOLCHAIN influences the nested build's toolchain selection the same way inherited RUSTFLAGS influences its flags, so it is fingerprinted the same way, as a parameter the session reads from the environment.

The walk now uses a private source manager, so computing the cache path no longer interns every closure manifest into the compilation session's source manager as a side effect.

The remaining walk-versus-resolver deviations are consolidated into one comment pointing at the closest in-tree sibling (frontend/masm's collect_dependency_metadata_for_scheme): path dependencies are extension-classified before canonicalization, and git declarations are recorded but never recursed. The module docs now cover the degraded cases and their recovery: Cargo-only projects without a miden-project.toml (root manifests hashed, no dependency recursion, reported at debug level), moved unpinned git revisions and transitive git dependencies being outside the closure, dropped-but-cached names lingering until the fingerprint next rotates, and the per-member-session assumption workspace builds rely on, noted where the root session is created.
The cargo-miden cache tests had become tautological: the lookup helper only returned directories that already contained the expected package, and the pre-build cleanup that attributed the artifact to the build under test was removed with the fingerprint layout. The tests now snapshot the time before the build and assert the located package was written at or after it, the helper accepts only fingerprint-shaped directory names so a regression to a flat layout fails, and the masm test derives the expected file name from the dependency name instead of repeating a literal.

The end-to-end FPI test previously covered only the same-fingerprint half of the design: a dependency source change rewrites the package in place and include_bytes! re-expands the consumer. A third build phase now covers the rotation half that #1302 is actually about: bumping the dependency's version in its manifests moves the cache to a new fingerprint directory, removes the old one, and the consumer's assembly carries only the new procedure root. The digest-recognition helper documents its coupling to the current u64-immediate lowering shape so a codegen change there is not misread as a stale-root regression.

Also guards the swapp-note fixture mutation like the existing one, and corrects the persist_cargo_miden_dependency docs: that directory is a legacy fallback consulted only when MIDENC_PACKAGE_CACHE is unset.
… publication atomic

The third review round found the lock protocol violating its own contract on two legs. A builder observing WouldBlock — meaning a pruner was deleting its directory at that moment — continued unlocked for the whole build, and the constructor doc claimed a lock it did not hold. And because flock binds to the inode, unlinking lock files after pruning let a builder hold a lock on an unlinked inode while the next pruner locked a fresh file at the same path and deleted the live directory.

Builders now take the blocking lock_shared on their fingerprint lock: pruners only ever try exclusive locks and never wait while holding one, and a builder waits only for its own lock while holding none, so the wait is deadlock-free and bounded by one in-progress removal. Lock files become permanent rendezvous objects — empty, bounded by the number of distinct fingerprints ever seen — which removes the inode ABA together with the three orphan-lock helpers. Preparation failures keep the cache configured so the first publication reports the concrete filesystem error, and the docs now describe that degraded mode instead of contradicting it, including that ownership is checked lexically by design and symlinked cache layouts are outside the contract.

Package publication writes to a process-unique temp file and renames over the target, so a concurrent identical-fingerprint build can no longer expose a truncated package to a reader; the remaining read-versus-include_bytes window is documented as part of the same-fingerprint boundary.

The fingerprint gains CARGO_ENCODED_RUSTFLAGS, which takes precedence over RUSTFLAGS when Cargo invokes rustc. The cache-layout machinery moves out of registry.rs into package_cache.rs, split into guard, create, lock, and sweep helpers, and the module docs now record what the memoized derivation assumes (root-session-only use, clones keeping their fingerprint, the deliberate target-dir exclusion), why the walk cannot reuse miden-project's resolver (it needs the registry whose cache path is being derived, and graph building performs git checkouts), which ambient inputs stay unfingerprinted and why, and the intended content-addressed end-state that belongs with the #1290 package redesign.
The compiler-side cache writers and pruner match the .masp extension case-insensitively via Package::EXTENSION, but the macro-side reader filtered with a case-sensitive literal — so on default-case-insensitive filesystems the pruner would delete a Foo.MASP the reader could never have resolved. Align the reader on the same rule.
…sserts

A new fixture pins the load-bearing invalidation mechanism by itself: two prepopulated cache directories whose basic-wallet packages embed different receive-asset roots, and two plain cargo builds of an unchanged consumer sharing one target directory, differing only in the MIDENC_PACKAGE_CACHE value. The embedded roots must follow the environment value in both directions — proof that the option_env! recorded by FPI expansions re-expands consumers on cache rotation without any help from manifest changes or midenc's own driver.

The build-attribution asserts in the cargo-miden tests tolerate whole-second mtime truncation, and the fingerprint-directory helper documents its newest-mtime tie-break. The integration-network compile_rust_package helper no longer persists packages to target/miden/<profile>: nothing under that suite reads the path, and compiler-driven builds resolve dependencies exclusively through the fingerprinted cache.
…cargo builds

Cargo prefers CARGO_ENCODED_RUSTFLAGS over RUSTFLAGS, and the nested cargo inherited the caller's environment — so a CI image or build-script context exporting the encoded variable silently replaced every mandatory Miden flag: no --cfg miden, no wasm target features, no immediate-abort panic strategy, with nothing attributing the resulting breakage to the inherited variable.

cargo_env now emits the composed flags in both spellings; the explicit encoded value makes any inherited one inert. The encoding splits the composed string on whitespace, which is exactly how cargo interprets the plain variable, so the flags cannot change meaning between the two forms.
Two preparation legs degraded harder than the documented contract. A lock-open failure aborted preparation entirely while its log claimed the build was continuing — the cache directory then materialized anyway through the first publication's create_dir_all, unlocked and unswept. And a cache-create failure returned None after the shared lock was already acquired, releasing the one guard that would protect the directory a later publication recreates. The first leg now creates the directory before returning, and the second keeps and returns the held lock, skipping only the sweep; the function doc states what Some and None actually mean.

With the encoded rustflags now set authoritatively for nested builds, the inherited CARGO_ENCODED_RUSTFLAGS value has no effect on what gets built, so it leaves the fingerprint (a comment records why, so it is not re-added).

The cache-path producer and the owned-layout validator were two unlinked lexical derivations; the parent-path construction moves next to the validator and the session derivation test asserts the produced path satisfies the ownership check, so a future layout change breaks a test instead of silently disabling locking and pruning.
…uites

Compiler-driven builds always set MIDENC_PACKAGE_CACHE and the FPI macro has no fallback once it is set, so persisting compiled dependency packages to target/miden/release never influenced any test — the five calls only wrote artifacts into the checked-in example and fixture trees. An editor-driven consumer expansion without the variable is served by the dependency's own cargo miden build output, not by test-suite side effects. The counter-note test also loses its dependency pre-build, which existed only to feed the persistence; the consumer build compiles the contract itself.
The option_env!(MIDENC_PACKAGE_CACHE) recording is the most user-visible behavior change on this branch — consumer crates now recompile whenever the fingerprinted cache path rotates — and it was missing from the changelog entries for #1302.
@greenhat
greenhat changed the base branch from next to i1302-MPC-unque-by-inputs August 6, 2026 13:21
@greenhat greenhat changed the title feat: embed component WIT in the compiled Miden package [2/2] feat: embed component WIT in the compiled Miden package Aug 6, 2026
greenhat added 10 commits August 6, 2026 17:04
The WIT generated by `#[component]` was written to `target/generated-wit/` and re-read by dependent crates' macros through `wit = "..."` path metadata in `miden-project.toml`, fragile plumbing every consuming project had to carry.

Embed the public WIT in a `rodata,miden_wit` Wasm custom section instead (with a linker uniqueness guard so two components in one binary fail at link time rather than concatenating into garbage WIT), carry it through the compiler pipeline alongside the account component metadata, and attach it to the `.masp` as a custom `wit` section. The `#[account(...)]`, sibling `#[component(pkg::Iface)]`, `#[note]`, and `#[tx_script]` macros now read dependency WIT from the dependency's compiled package, so the `wit` keys are no longer read, prebuilt `.masp` file dependencies are self-contained, and packages without embedded WIT are rejected with a rebuild hint. Components authored manually (a local `wit/` directory with a bare `miden::generate!()`) embed their single WIT file the same way.

The WIT section is not covered by the package content digest until miden-mast-package gains a first-class WIT section id upstream. The uniqueness guard adds one exported data byte, which shifts expected package sizes and VM cycle counts slightly.
The account component metadata and the component WIT were threaded through the compiler as two parallel `Option<Vec<u8>>` fields, so every additional out-of-band payload would have to be added to each stage struct, every construction site, and the growing `assemble_with_registry` signature.

Introduce `PackageSections` in `midenc-frontend-wasm-metadata` (next to the section-name constants it complements) and carry it as a single field through `FrontendOutput`, `MidenComponent`, and `CodegenOutput`; assembly takes `&PackageSections` and attaches all payloads in one `attach_package_sections`. Adding a future payload is now one field plus its producer and consumer — stage signatures and construction sites stay fixed. `ParsedModule` keeps its borrowed per-section slices; the carrier starts where the data becomes owned.
…re local WIT

Review of the WIT-in-package branch surfaced three defects. The uniqueness-guard export emitted next to the WIT section was the feature's only executable-data footprint: one `#[used]` data byte shifted rodata and cost ~10 VM cycles on every transaction, and metadata must not perturb data segments, cycle counts, or commitments. The wit-bindgen resolver loaded the crate's local `wit/` directory before the WIT embedded in dependency packages, so a manually authored component whose local WIT imports a Miden dependency failed with a bare "package not found". And core Wasm module inputs parsed the metadata custom sections but dropped them instead of attaching them to the package.

Drop the guard and detect the failure it guarded against in the frontend instead: linking two `#[component]` implementations concatenates their identically named custom sections, which now surfaces as a diagnostic counting top-level `package ...;` declarations in the section (the duplicate-sections assert in the component translator becomes a diagnostic as well). Package-size and cycle expectations revert to their pre-branch values, confirming WIT embedding no longer touches runtime state. Load WIT sources in dependency order — SDK prelude, then dependency packages, then the local `wit/` directory — and thread `PackageSections` out of `translate_module_as_component` so core-module inputs carry their sections too.

Also reword stale diagnostics and docs that referenced the removed WIT path metadata (including the unreachable empty-paths error in `generate!`), name the single-`.wit`-file rule for manually authored components in the missing-WIT error and MIGRATION.md, and add regression tests for the concatenation detector and for local WIT importing a dependency package.
…kage lookup

A second review round surfaced holes in the WIT embedding and the `.masp` lookup. Bare `miden::generate!()` embedded the local WIT file verbatim, imports included — but consumers resolve embedded WIT against the bundled SDK WIT alone, so such a package failed downstream with a misleading "rebuild the dependency" hint. The package search consulted ambient directories (`CARGO_TARGET_DIR`, `OUT_DIR`, cwd targets) with a name-blind solitary-`.masp` fallback, which could silently bind the wrong package and made the missing-package tests fail when a stray workspace artifact shared the fixture's name. The concatenated-section detector hard-errored on valid WIT with block-commented `package` lines and missed true concatenation when a blob lacked a trailing newline, and a repeated same-named metadata section within one module silently overwrote the first.

Producers now parse the candidate local WIT against the SDK prelude and skip embedding when it is not self-contained or exports nothing, routing consumers to the accurate "does not embed component WIT" error; the consumer diagnostic names the self-containment requirement when embedded WIT references a package that is not embedded alongside it. Package lookup prefers the freshest name-matched artifact across profile directories (Cargo never sets `PROFILE` for proc macros, so profile order alone lets a stale debug package shadow a fresh release build) and accepts a solitary `.masp` only in the dependency's own target directories; the unit fixtures use fixture-unique dependency names.

Embedded WIT payloads are wrapped in boundary newlines so section concatenation always keeps `package` declarations on their own lines, and the declaration counter strips nested `/* */` block comments. Both the account-metadata and WIT section arms now reject a repeated section per core module, and the translator merges all package-section payloads through one `collect_package_sections` helper with a uniform at-most-one-module error, replacing the per-payload gather-and-assert blocks.
The dependency package search could still adopt the wrong artifact: the solitary-`.masp` fallback treated shared ancestor target directories (a workspace's `target/`, which holds every member's packages) as private to the dependency, stem matching returned the first alias per directory so a stale Cargo-named `dep_fixture.masp` shadowed a fresh `dep-fixture.masp` regardless of age, and ambient directories never got the freshest-match rule at all.

Deserialize every candidate found by searching and accept it only when its package id matches the dependency's name (normalizing hyphens/underscores); rejected candidates are listed in the not-found error. Order name matches freshest-first across profile directories and stem aliases — own directories before ambient — and confine the name-blind solitary fallback to the dependency's private `<root>/target` directories. An explicit `.masp` file dependency remains the manifest's choice and skips the id check, since the manifest key need not equal the prebuilt package's id. The shared `read_package` helper also replaces the duplicated reader in the FPI flow, and the package fixture writer moves to a common test-support module.

Also require exactly one top-level WIT package declaration in the embedded section (a zero-declaration payload now fails at the producing crate instead of in a consumer), reuse the section merge helper for core-module inputs, name the exports-an-interface requirement in the missing-WIT error, and document the editor workflow in MIGRATION.md: `cargo check` of a dependency no longer regenerates its WIT as a side effect, so dependencies need one `cargo miden build` before checking dependents.
The concatenation detector disqualified any line containing a `{`, so a valid whitespace-insensitive declaration like `package miden:x@1.0.0; interface api { ... }` counted as zero packages. Such WIT passes the producer's real-parser self-containment check and gets embedded, after which the exactly-one validation failed the whole build with a misleading "does not contain a top-level WIT package declaration" — and two concatenated one-line-style packages produced the same wrong message instead of the dedicated duplicate-implementation diagnostic.

Reject a `{` only when it appears before the first `;`, which still excludes nested `package <id> { ... }` declarations. Also wrap the cross-module duplicate-section error in the typed `WasmError::Unsupported` used by the neighboring frontend-metadata merge instead of a bare report.
Dependency package resolution verified the package id but ignored a `Path { version: ... }` manifest pin, so a right-name/wrong-version artifact was adopted silently, and a format-skewed `.masp` (the likely failure after a toolchain upgrade) surfaced a bare deserialization error with no action attached. The package deserialized for the id check was also dropped and re-read from disk by the FPI flow, wasting a full MAST-forest decode and extracting procedure roots from a different read than the one that was identity-checked.

Check semantic and exact version pins against the candidate's version during resolution — digest pins stay with the assembler, which enforces them at link time — and fold both id and version rejections into the not-found diagnostic. Append the rebuild-with-current-toolchain guidance to deserialization failures. Carry the resolved package (now an `Arc`) through `DependencyWitSource`/`SelectedDependency` so FPI extracts procedure roots from the verified read instead of re-reading the file.

Also drop the dependency's own target directories from the shared ancestor list (the ancestor walks start at the root, re-discovering them, so each was scanned and reported twice), split the package-fixture builder so tests can construct in-memory packages, and retire the stale `midenc-fpi-` fixture prefix left over from the code's previous home.
…tant

The account-component metadata section name was a bare "rodata,miden_account" literal repeated at the producing macro, the frontend match guard, its duplicate-section diagnostic, and the cross-module merge label, while its sibling WIT section already had a shared constant — the exact producer/consumer drift the constants exist to prevent.

Add `WASM_ACCOUNT_COMPONENT_METADATA_CUSTOM_SECTION_NAME` to `midenc-frontend-wasm-metadata` next to the other section names and use it at every site; the literal now exists only in the constant's definition.
…embedded WIT

The package.metadata.miden.dependencies.<name>.wit key in
miden-project.toml is consulted only when the dependency package has no
embedded WIT section, as an escape hatch for packages produced by
toolchains that do not embed WIT. The key may name a .wit file or a
directory containing exactly one top-level .wit file, and the override
must satisfy the same self-containment rule as embedded WIT. Setting the
key for a package that embeds WIT is an error, and the .masp package
remains required in all cases.
The MIDENC_EMIT_WIT public-WIT dump read target/generated-wit, which this branch no longer produces, so the helper silently did nothing while the README kept documenting the feature.

Extract the wit section from the compiled package instead, and move the hook from the Cargo-fixture builder (where no package exists yet) into CompilerTest::compile, right after the package is stored. The dump now fires for every route that assembles a package, is named per artifact, and skips packages that embed no WIT; the stale generated-wit timing comment at the old call site is gone.
…n cargo builds

Builds that midenc does not drive (cargo check, cargo build, IDE analysis)
expand the SDK macros without a populated package cache, so dependency
packages could not be resolved and the editor showed errors (#1215).

Every contract template and example now ships a std-only build.rs. Outside
a midenc-driven build it locates the project's fingerprinted cache with the
new `cargo miden package-cache` subcommand, populates it with a nested
`cargo miden build --release` when the project has source dependencies, and
exports MIDENC_PACKAGE_CACHE to the crate's macro expansion. Inside a
midenc-driven build the inherited variable short-circuits the script, which
is also the recursion guard. The nested build runs against a dedicated
cargo target directory because the outer cargo holds its build-dir lock
while build scripts run.

The subcommand prints the cache directory, the number of dependencies
compiled into the cache, and the watch paths a build script must observe,
backed by the new `Session::package_cache_build_inputs` API. The watch list
covers the manifest closure, dependency sources, and the cache directory
itself, so dependency edits, compiler updates, and cache pruning re-run the
script while root-source edits do not.

The template miden-project.toml manifests also gain the `[lib].path` key
the VM v0.25 project model requires; without it, projects generated from
the templates failed to parse in both driven builds and macro expansion.
The cargo-expand dump helper now passes the session's cache path so
expansion resolves the same packages the driven build used.

Closes #1298
…SDK macros

The macro-side resolver had grown its own dependency discovery: it walked
the dependency's, enclosing workspaces', and ambient target/miden/<profile>
directories, picked the freshest stem match, adopted solitary packages, and
re-checked package ids and version pins. With the package cache fingerprinted
by build inputs, rewritten by every build, and exported to every macro
expansion by midenc-driven builds and the contract build script, that
machinery duplicated the compiler's dependency management and could observe
artifacts the current build never produced.

BREAKING: resolution now has exactly two paths. A manifest path that names a
`.masp` file is read from that location, with no name matching, so renamed
prebuilt packages from other toolchains keep working. Every other dependency
is read from the `MIDENC_PACKAGE_CACHE` directory under its package name,
trying the hyphen and underscore stem spellings, and the found package is
trusted as-is; id, version, and digest verification belong to the compiler's
project resolution and the assembler. Without a configured cache, expansion
fails with instructions to build through `cargo miden build` or to add the
contract `build.rs`, instead of searching the filesystem.

Unit tests point resolution at per-fixture caches through a thread-local
override, since the process environment is shared across parallel tests. The
sibling-component test harness publishes its synthesized package into a
project-local cache directory and exports the variable to its builds.

Closes the discovery-cleanup follow-up of #1298
The `release lint` CI job verifies the checked-in
tools/cargo-miden/templates.tar.gz against the extra/templates sources.
The #1298 changes added a build.rs to every contract template and the
`[lib].path` keys to the template manifests without regenerating the
bundle, so the lint reported it as stale.
@greenhat
greenhat force-pushed the i1302-MPC-unque-by-inputs branch from 104f97e to a653845 Compare August 10, 2026 09:56
…solution

Resolution correctness:
- The `.wit` file selected by a `wit` manifest-key override is recorded on
  the dependency source and registered as a build input alongside the
  `.masp` path. In the override flow that file is the only source of the
  dependency's interface, and editing it previously changed nothing rustc
  tracks, so consumers kept stale bindings.
- The cache prober now reads `miden-project.toml`'s `[package].name` as its
  first filename stem. The cache writer names files after the Miden package
  name, which the prober never tried; resolution succeeded only when the
  Cargo name, manifest key, or directory name happened to coincide.
- Each dependency package file is deserialized once and reused: a
  thread-local memo keyed by path and validated by modification time and
  length serves the repeated reads one expansion performs from its several
  entry points. This also stops a concurrent cache rewrite between split
  reads from pairing generated bindings with procedure roots of different
  package generations, and the revalidation keeps long-lived proc-macro
  hosts (rust-analyzer) from pinning stale packages.

Diagnostics:
- A malformed `package.metadata.miden.dependencies` shape (non-table
  levels) is a hard error instead of being treated as an absent key, which
  silently disabled the override and bypassed the embedded-WIT conflict
  check.
- A macro reference to a dependency declared with a workspace, workspace
  path, or git scheme names the unsupported scheme instead of claiming the
  dependency is not declared; the scheme match in collection is exhaustive
  so a future scheme cannot be skipped silently.
- When every exported WIT interface is skipped (unversioned package,
  anonymous export), the collected skip reasons are appended to the "no
  exported WIT interface found" error instead of being discarded.

The test-only package-cache override now resets on panic, so a failed
assertion cannot leak the override into another test on the same thread.
…e manifest walk

The build-script input collector watched a hard-coded `src` directory per
dependency project, while `[lib].path` is a required, configurable manifest
key: a dependency whose target sources live elsewhere never re-triggered
the nested build, and consumers kept stale packages. Each loaded project's
declared target source directories are watched instead (falling back to
the file itself when the source sits directly in the project root, which
would otherwise sweep `target/` churn into the watch set).

The collector also had no knowledge of `wit` manifest-key override files,
the only interface source of the override flow; every visited project's
`package.metadata.miden.dependencies.<name>.wit` entries are now watched,
including the root's.

The source-dependency count moves into the walk itself, counted where each
direct root dependency is classified. The previous standalone counter
re-implemented the scheme classification, skipped the walk's non-`file`
URI guard, and loaded the root project a second time.
… consumers

The fallible section-id construction with its identical expect message was
hand-rolled in five places, and the find-section-then-decode sequence was
copied three times. `midenc-frontend-wasm-metadata` owns the section-id
constant and is already a dependency of every consumer, so it now exports
`package_wit_section_id()` and `package_wit(&Package)`, giving the planned
upstream `SectionId::WIT` migration a single place to change.
…ith the producer

The frontend's duplicate-`#[component]` detector required the literal
`package ` (one ASCII space) and `;` on one line before any `{`. A tab
after the keyword, a declaration split across lines, or a block comment
between the tokens made valid WIT count as zero declarations — a hard
error on source the macro side had just validated with a real WIT
resolver — and two declarations on one line counted once. The scan is now
token-based with word boundaries, a closed block comment leaves a space so
tokens it separated do not fuse, and byte-wise glued sections are detected
even without the boundary newlines the producers add.

The scanner moves to `midenc-frontend-wasm-metadata`, whose constants
already define the section contract, and `generate_wit_link_section`
debug-asserts the producing half — every embedded payload holds exactly
one top-level declaration — pinning the cross-crate contract at both ends.

`merge_section_payload` now reports the duplicate-section error through
the diagnostics handler like the sibling checks, instead of a
`WasmError::Unsupported` that rendered as "unsupported WebAssembly code".
…the package cache

- `cargo miden build` and `cargo miden package-cache` share one
  session-construction path. The contract build script depends on both
  commands deriving the identical fingerprint from the same arguments, so
  the bootstrap they must agree on now exists once.
- The package-cache liveness-lock path is exposed (hidden) from
  `midenc-session`; the p2id-note test joins the liveness protocol through
  it instead of re-deriving the private lock naming.
- The build-script identity test discovers contract templates and scaffold
  contracts instead of hardcoding them, so a future template cannot ship a
  divergent or missing `build.rs` unnoticed.
- The `.wit` directory enumeration shared by the override reader and the
  local-world detector lives in one helper.
- `ResolvedWit.paths` becomes `prelude_dir`: the vector always held exactly
  the SDK prelude directory. Resolver internals
  (`resolve_dependency_package`, `read_package`,
  `ResolvedDependencyPackage`) are module-private, and FPI's
  `load_dependency` destructures its owned dependency instead of cloning
  out of it.
… behavior

- The generated project's rust-sdk-patterns skill instructed users to
  declare cross-component dependencies through the removed Cargo.toml
  metadata tables and pointed at a manifest that no longer contains them;
  it now shows the miden-project.toml `[dependencies]` key.
- The swapp-note fixture drops its dead `[package.metadata.miden.dependencies]`
  and `[package.metadata.component.target.dependencies]` tables, whose only
  reader is gone.
- The build script documents the accepted stale-cache window of a failing
  nested build, and MIGRATION states both transient failure modes of the
  plain-cargo handoff (stale packages after a broken dependency build,
  cache pruning by a concurrent driven build) next to the adoption steps.
  The editor-workflow note now mentions that the build script automates
  the manual dependency rebuild.
- The `manifest_paths` module doc no longer claims Cargo.toml metadata
  resolution, and the assembly module doc names the `.hir`-input caveat:
  recompiling an emitted HIR artifact yields a package without the
  account-metadata and WIT sections, since there are no Wasm custom
  sections to extract.
- The embedded template bundle is regenerated for the skill and build
  script changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explore using build.rs in the contract to compile dependencies Use account's Miden package WIT file in a note script project

2 participants